In an effort to reduce monitoring noise, we've set up a simple Powershell script to throw hosts into maintenance mode prior to patching and deployment.  Unfortunately we are seeing that while this prevents notification on the initial creation of the alert, we still get notifications (most of the time) when the alerts are closed - even if the host is still in maintenance mode.

The "comment" for these closed alerts are:  "Auto-resolving alert: entity in maintenance mode."

Clearly SCOM knows the host is in maintenance mode, so my question is - why is it generating the alert and notifying on it - and what can we do to prevent it?  If there isn't an easy way to prevent it - can we potentially set up our alerts to only fire when the closing user is NOT "Maintenance Mode"?

For reference the alerts that are generating are simple Windows service monitoring alerts.

As far as I know, in maintenance mode, alerts, notifications, rules, monitors, automatic responses, state changes, and new alerts are suppressed at the agent.

But you may also get alert when the computer is in maintenance mode. Because when a computer is put into maintenance mode, the database and the management server are updated immediately.  At that point, any server side responses including notification rules will not run.  The agent will be polling the MS on its polling interval (default = 5 min.) as usual looking for configuration updates.  Until the agent makes a configuration update, it will be unaware that it should be in maintenance.  It will continue to generate alerts and run responses until it gets a configuration update.  So - on average, by default an agent will continue to generate alerts for 2.5 minutes after being put into maintenance mode.  Similarly, on the flip side when an agent is taken out of maintenance mode, on average it will be 2.5 minutes before it starts to generate alerts and run responses again.

Regards,

Yan Li

Thanks for the response!

I don't believe this is a case of agents not being fully in or out of maintenance mode.  We get these alerts EVERY time we put a host into maintenance mode and take them out.  We have scripted our deployments to put the host into maintenance mode for 30 minutes.  The actual service restarts happen probably at or around 10 minutes in and are well done before the end of the maintenance window in SCOM.  The alert clear messages happen immediately after the host exits maintenance mode every time (even if the services had restarted 15-20 minutes prior).  We will get 10 service restarted closed alerts at the exact same interval even if they restarted at different times during the window.

This seems more related to HOW the hosts are put into maintenance mode ("not monitoring" the host) and then coming back.

I can clearly see that alerts are generated and closed for every service that is restarted while the host is under maintenance mode.  These aren't visible in the normal console, but if you open up a custom view that shows closed alerts - that is where I am seeing that the alert was auto-closed by maintenance mode.

So 2 questions still:

1) How can we fix this?  This doesn't seem like it is intended behavior, or at least is not consistent with any other monitoring product I am familiar with.

2) If there isn't a clear fix, can we somehow modify our notification rules to NOT fire when "Maintenance Mode" is the closer?  I don't see a way to do this in the GUI, but perhaps manual editing of the XML source could accomplish it?

Hi

I suspect your subscriptions are set to "delay" and also to trigger for closed alerts - this means that they will fire after alert has been in a given resolution state for more than the delay configured. It is a "feature".

Workaround:

http://www.maartendamen.com/2009/12/scom-alert-notification-subscription-delay-sending-for-x-minutes-and-dont-sent-if-alert-is-auto-resolved-within-that-time/

Cheers

Graham

@Take 2 - we aren't currently delaying notifications.  We don't get any notifications for the initial "problem" notification (the host is under maintenance mode at the time) but we do get the closed notifications.

I suspect this particular workaround would not do what it is that we need, although it does bring to light another possible workaround scenario using Powershell to send the email alerts.

I found the resolution to this problem.  I just experienced it as well.  Are you putting your Management servers in Maintenance mode?  If so this is your problem.  I screwed up on our last maintenance period and accidentally put our Management servers in Maintenance mode.  When i pulled them out is the exact time stamp on all of the alerts that auto closed with the response of "Auto-resolving alert: entity in maintenance mode.".